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Definition and Scope 


• Rehearsals are mission level readiness tests that exercise 
personnel, operational process, and flight products, in a near flight- 
like environment. 

- The program is started 6-9 months prior to launch and is used to ensure 
the final as built system will meet mission goals (i.e. validation) 

- On Deep Space missions you rehearse cruise activities post launch! 

• Focus on critical activities to the mission, (i.e. propulsive maneuvers, 
instrument commissioning and any first time events or coordinating 
activities that involve major stakeholders): 

- Examples: 

• Solar Array deploy, orbit insertion, planet fly-by 

• Delta-V, Trajectory Correction Maneuvers, Gyro/Star Tracker caiibration 

• Instrument power on, deployment and initial configuration/calibration 

• Collision Avoidance process 

• Shift handover 
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Rehearsal Program Phases 


• Plan (“Development”) 

- Rehearsal Director produces the Mission Rehearsal Plan; 

- Provides specifics on the planning process and conduct process for the 
exercises; 

• Execute (“Simulation Exercise”) 

- Bulk of work: Rehearsal Working Group, Simulation Team meetings, NASA 
Reviews, internal reporting to Project 

- Anomaly development, dry runs, internal timelines finalized 

- Kickoff briefing to focus team of expectations and objectives for exercise 

- Exercise the team 

• Keep good notes! 

• Follow Up 

- Debrief team (Hot Wash and Formal Team Debrief) 

- Track, re-assign and closeout actions 

- Update the plan per feedback 

- Generate Post Event Report - Assessment of Team based on debrief 


Rehearsing 101 


4 


Plan (1/2) 


• Develop a Mission Rehearsal Plan 

- Typically 6-12 months before the first exercise 

- Initial draft (high level) for the content of each exercise 

• Timelines to be exercised 

• Select the Level 2 or 3 requirements for Validation 

• Products and teams needed 

- Initial schedule and number of exercises 

• Reused for MOR charts! 

- Best level for document: Explain the PLANNING process and 
CONDUCT for the exercises 

• Define the key stakeholders and interfaces to be exercised 

- Boundaries on how much will be configured flight like may be 
established here OR during Rehearsal forum 

- Have a good End to End System Diagram to reference 
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Plan (2/2) 


• Develop the roles for the Rehearsal Director and Simulation (“sim”) 
Team 

- Rehearsal Director works with the Project and directs Simulation 
Engineer 

- Individuals with knowledge across Systems Engineering, Ground 
System Development, Spacecraft Operations, Planning, Networks, 
Navigation, Science Operations, and flight timeline 

- Relationship with team leads ensures unique mission aspects are 
communicated and prioritized 

• Develop the Flight Team Organization 

- Who are the players? 

- How will they communicate? 

- Where will they reside for the exercise? 

• Anomaly Process to follow 

- Summarize the operational anomaly process to be exercised 

- Identify the system to be used 
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Execute 


• Rehearsal Working Group: 

- Forum to discuss scope of exercise; upcoming schedule; availability of 
personnel; poll stakeholders on readiness; select timeline to rehearse; 
refine console organization; communicate objectives and success 
criteria; and track actions on any obstacles to rehearsing 

• Simulation Team meetings: 

- Discuss sim configurations; script CM; boot up procedures; plans during 
a reset; green card communication; dry run plans (i.e. ground versions) 

- Main topic : Nominal timeline and anomaly development to ensure team 
is adequately trained for off-nominal scenarios; 

- Plans for coordinating actual exercise (i.e. internal timeline, red/green/ 
brown cards) 

• Kickoff (“crew briefing”) to focus team on timeline and objectives 
for exercise; provide key initial conditions 

- Detail the when, where, who and what for the exercise (“console times”) 

• Exercise the team! 


Rehearsing 101 


7 


Core Objectives 


• Common high level objectives are: 

- Exercise mission timeline using operational processes and flight 
products: 

- Exercise the mission timeline; 

- Train on communications system and exercise voice protocols; 

- Exercise team interaction during nominal and contingency situations; 

- Exercise anomaly identification and resolution process; 

- Train on the flight configured ground system; 

- Exercise with system hardware configurations; 

• Objectives are unique based on rehearsal content 

- Communicate early and often during rehearsal meetings with 
stakeholders; modify as appropriate 
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When Things Do Not Go As Planned 


• Advice on how to manage unplanned events during an exercise 

- If the simulation stops, brief the team to continue working pre-existing 
issues 

• Decide with input from the Simulation Team the options; brief the team on 
what issues continue to be valid and those that can be disregarded 

- If a reboot is necessary, decide with Flight Management if it is 
worthwhile to continue or release the team 

• Assess the situation quickly 

- Determine if it’s negative training (i.e. team is moving down a path just 
to keep the simulation moving forward) 

- Avoid non-flight like operations or procedures to take place; 

- If a simulation reboot is necessary, coordinate the new start time; 

- Consider if a new or slightly different objective(s) is playing out; 

• Is there a valuable training opportunity to be gained? 

• Communicate direction and instructions to keep the exercise moving 
BUT avoid any judgment calls for the teams: 

- Explain the new conditions and move forward 

- Follow up with any additional green cards 
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Anomaly Development 


• Develop in a spreadsheet 

• Iterate with the Simulation engineer 

- Ask for further analysis or investigation on proposed anomalies; 

- Prioritize (1=must have, 2=interesting, 3=time permitting); 

- Better to know what you want, that the simulator is able to model this 
and that it is credible before deciding to dry run; 

- Plan on several weeks of running down anomalies and prioritizing; 

- When final, place into an internal detailed timeline for the simulation 
team to follow during the dry run and finalize for the formal exercise; 

- Remember: There is always ways to trip up operations - pick the best 
OR research prior failure reports!! 

- Consider using a theme (i.e. points to be stressed) for the exercise; 

• Perhaps this team is too concerned with communication during first time 
events 
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Sample Anomaly Development Sheet 


Anom. 

Order 

ID 

Crit 

Title 

When? 

Cause 

Rationale 

Liens 

Notes 

4/8/14 



1 











Post MOI 
Phase set / 1C 



How much will NAV OD solutions be 
affected? What if a RW failed? 


1 

15 


S/C enters Safemode 

RW failure 



2 

, 

1 

bMSA needs reboot prior 
to MOI-24 hours 

MOI-2 days 

Network File Server issue 

Exercise process for bMSA 
(checks) to take command control 
from MSA; How long to try MSA 
before deciding switch to bMSA? 

Does NFS also affect bMSA? Could 
be power outage...Further scheming 
with dA on how to accomplish 
this.... 

Consider bMSA goes down to 
stress SA...and force the process 
for getting the bMSA up and 
running and ready to support 
(checks); then got to it for 5a 

3 

4 

2 

NAV data suspect - need 
alternative method for 
resolving OD 

Prior to MOI- 
24 hours 

Incomplete or unexplained 
Doppler, Range and DDOR data 

Poke at the prime and backup 
methods for resolving OD with the 
NAG 

Looking into ways where data can 
become corrupted and/or 
unavailable from DSN pov; May 
need to provide some "wacky" OD 
solution plots or scheme with 
SD/Tomas to see how to make this 
beneficial.... Tomas will provide data 
with inconsistent results (5/13) 

Include BU method here 
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Follow Up 


• At the end of the exercise do a Hot Wash; 

• Provides immediate Rehearsal Director feedback on what was 
observed and needs work; 

- Never single anyone out; 

- Keep it as constructive criticism; 

- Lead with positives before communicating the negative; 

- No formal actions taken or worked. 

• Follow within a week with a Formal Team Debrief; 

- Discuss key areas for improvement (“relive the key moments”); 

- Assign formal actions; 

- Track, re-assign and closeout actions at appropriate working group 

• Update planning or simulation per feedback and experience; 
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Challenges and Workarounds (1/2) 


• Low fidelity simulation environment 

- Communication, thermal, power, instruments 

- Document (DR) the issue; advocate for fix; use green/red card 

• Too few opportunities to exercise critical activities 

- Minimum set 2 (one nominal; one anomalous) if operations team is 
mature 

- Maximum set 5 (or more) if operations team is newly formed or finely 
tuned operations is necessary 

- Seek guidance on Project for what has been contracted; advocate to 
change (if needed) 

• Process or Product maturity 

- Awesome opportunity; “threat” of rehearsal will focus team on 
developing 

- Monitor progress for process or product development 

- Status in Rehearsal forum; assign actions; highlight or modify objectives 
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Challenges and Workarounds (2/2) 


• Personnel availability 

- Keep a list of who shows up 

- Advertise the minimum TWO for critical or first time activities; work 
concerns with Project 

• Simulation exercise dry run time 

- State up front in the Rehearsal plan what is expected; work with 
simulation engineer 

- If an issue, report and work with Project for time needed 

• Moving rehearsal date 

- Project dictates based on personnel and scheduling 

- Rehearsal plans should be in place to go as early as 2 weeks prior 
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Lessons Learned (1/2) 


• Know your mission 

- Small, medium, large 

- How many rehearsals are in the contract? 

- Are all stakeholders contracted to support? 

- How mature are the interfaces? 

- How mature are the teams? 

- Have they supported NASA before? 

• Develop console organization early 

- Engage Mission Manager on down; do 3-4 months before first rehearsal 

- Work the roles and responsibilities of the team and individuals 

- Develop a room layout and have everyone buy off 

- Good idea to do Launch exercise first in order to flush out these details 
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Lessons Learned (2/2) 


• Focus on the critical activities in the timeline 

- Independently assess in case more exercises are necessary 

- Advocate for more time during a rehearsal opportunity (i.e. make 2 days 
3 days; 8 hours to 12 hours, etc); or 

- Add days to the schedule 

- Do early to get commitments and add to schedule 

• Assess the simulation environment 

- Request/develop a limitation list early; 

- Recommend critical simulation upgrades but stay within reason; monitor 
delivery work in terms of time taken away from dry running 

- If DSN is to route data, they need 3 months lead time to schedule. 

• Be prepared to rehearse earlier than expected 

- Stay focused on stakeholder readiness to exercise 

- Are processes mature? Products ready? 

- Develop minimum dry run time AND schedule! 
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Observations: LEO and Deep Space Missions (1/2) 


• Terminology 

- Operations Readiness Tests vs Rehearsals 

- Green Cards vs Red Cards 

- I&T vs ATLO (Assembly, Test and Launch Operations) 

- RTLT (Round Trip Light Time delay) 

• Project Schedule 

- Incompressible Test List vs Additional Ground Testing 

- Rehearsals post-launch (teams too busy on road to launch) 

• Scope of exercise and time allocated 

- Function of mission type (Scout vs Explorer vs Flagship) 

- Function of team experience 

• Flight Team Experience 

- Multi-mission, matrixed out; ACEs (Flight Controllers) have Masters 

- The size and experience of the team vary depending on the type of 
missions as well as on the budget and the contract constraints 

- Beware of overly complex systems and large teams as they need more 
inter-team coordination and process adherence 
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Observations: LEO and Deep Space Missions (2/2) 


• Systems Engineering and Life Cycle 

- I&T ops system is an advance copy of the flight system (“mini-MOC”) 

- GSFC I&T and Flight systems typically have been different 

• Hardware vendor is different than Operations team 

- Requirement verification and validation is large scope for large JPL 
missions; 

• Heritage requirements levied on operations team with little staff is challenging 

- Ground Readiness Tests are not visible for heritage systems (exception 
0-REX); 

• Product generation: 

- LEO: @Launch it’s FINAL (lower risk) 

- Deep Space: @Launch planned on-going development (higher risk) 

- Deep Space: High heritage tools are used in product development and 
I&T tests 

• Final tool release occurs prior to launch with final products generated 6-8 
weeks prior to spacecraft uplink 

- LEO: Final release of tools used to create final flight products pre-launch 

• No plan to develop products post-launch (except for contingencies) 
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Questions? 


